On-line image processing and communication system

ABSTRACT

An image data manipulation system is described in which users located remotely from an image data storage library may participate in a collaborative image data rendering and evaluation session. The system includes the exchange of state parameters between the client computer of a user controlling the image rendering, the session driver, and a server computer which relays updated state parameters to other client computers participating in a session. The state parameters are used to update the view on each users computer to keep all the displays of the participants in synch with that of the session driver. The server processes extensive image rendering task for which the remote clients are not equipped and transmits newly-processed image data to the clients as appropriate. One embodiment for educational applications utilizes pre-stored image data sets which eliminates the need to transmit large blocks of image data over a network during a collaborative session.

BACKGROUND OF THE INVENTION

[0001] The present invention generally relates to miniPACS (PictureArchiving and Communications System) or teleradiology systems,specifically to miniPACS/teleradiology systems with remote volume dataprocessing, visualization, and multi-user conferencing capability. Inour previous patent application, U.S. patent application Ser. No.09/434,088, we presented a miniPACS/teleradiology system with remotevolume data rendering and visualization capability. The presentinvention is directed to additional features and enhancements of thearchitecture described therein.

[0002] Teleradiology is a means for electronically transmittingradiographic patient images and consultative text from one location toanother. Teleradiology systems have been widely used by healthcareproviders to expand the geographic and/or time coverage of their serviceand to efficiently utilize the time of healthcare professionals withspecialty and subspecialty training and skills (e.g., radiologists). Theresult is improved healthcare service quality, decreased delivery time,and reduced costs.

[0003] One drawback to some existing teleradiology systems, however, isthe lack of the ability for radiologists to communicate interactivelywith their colleagues and referring physicians from disparate locationsfor the purpose of consultation, education, and collaborative studies.Collaboration is especially important for studies using volumetricimages, where the ability to interactively manipulate the volumetricimages and simultaneously view the processed images is essential forrapid and effective communications between multiple participantsinvolved.

[0004] There are numerous methods and systems providing multi-medianetwork based conferencing capability. However, these methods andsystems only support shared viewing of texts, documents, and videos.Furthermore, a radiology conferencing system presents unique obstacles.For example, the size of data to be transmitted could be very large andthe requirement on image (picture) quality could be very high. To beclinically useful, the transmission should be interactively “on-demand”in nature. There are ongoing efforts to develop radiology conferencingcapabilities for the communication of two-dimensional (2D) images.However, none of these systems supports interactive communication ofvolumetric/three-dimensional (3D) images.

[0005] As a result, there exists a need for a miniPACS/teleradiologysystem with network based conferencing capability supportingsynchronized distribution and viewing of interactively processedvolumetric images. Further, there exists a need for an improved methodand procedure for the management of multi-center trials involvingvolumetric images.

SUMMARY OF THE INVENTION

[0006] The present invention provides a computer architecture for aclient/server-based advanced image processing and rendering system. Thepresent invention further provides a computer architecture to supportmulti-user concurrent usage of the processing server. The presentinvention includes a method and apparatus that combines thenetwork-based conferencing capability with remote interactive advancedimage processing capability. The present invention enables users fromdisparate locations to interactively manipulate images andsimultaneously view the processed images in an independent orsynchronized fashion. The present invention further enables a user tointeractively view and manipulate the images without having to downloadthe entire volumetric data set. The present invention also includesimproved methods and procedures for radiology consultation andmulti-center trial management involving volumetric images using theabove-mentioned technology.

[0007] The present invention may be used for radiology consultation. Inone step, the acquisition of 2D or 3D/volumetric image/data sets orretrieval of previously acquired image/data sets is performed. Thevolumetric data set could be three-dimensional in space, or two- orthree-dimensional in space and one-dimensional in time, e.g.,time-resolved spatial data sets. In another step, data is moved to aserver, which could be the scanner workstation itself or a separatecomputer connected to a network, and which has the conferencingsoftware. In another step, client software is initiated by a remoteuser/users. Each user is able to remotely access and manipulate the 2Das well as volumetric/3D images with full processing capabilities,including Multi-planar Reformat (MPR), Maximum Intensity Projection(MIP), Volume Rendering, Image segmentation, and etc. As described inthe preferred embodiment, an user may send the image processing request,such as MPR request to the server, the server will render the imagesaccordingly and send the result back. In another step, each user is ableto interactively manipulate volumetric images without transferring theentire dataset, employing an “on-demand” image transmission method.

BRIEF DESCRIPTION OF THE DRAWINGS

[0008]FIG. 1 depicts a block diagram of the present invention.

[0009]FIG. 2 depicts an alternative diagram of the present invention.

[0010]FIG. 3 depicts a flowchart of method of the present invention.

[0011]FIG. 4 depicts a description of state parameters which may be usedin one embodiment.

[0012]FIG. 5 depicts a flowchart of the state parameter updating method.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT

[0013]FIG. 1 depicts the teleradiology system described in our previouspatent application, U.S. patent application Ser. No. 09/434,088. Theteleradiology system includes as data transmitting station 100, areceiving station 300, and a network 200 connecting the transmittingstation 100 and receiving station 300. The system may also include adata security system 34 which extends into the transmitting station 100,receiving station 300, and network 200. Receiving station 300 comprisesa data receiver 26, a send request 22, a user interface 32, a datadecompressor 28, a display system 30, a central processing system 24,and, data security 34. Transmitting station 100 comprises a datatransmitter 16, a receive request 20, a data compressor 14, a volumedata rendering generator 12, a central processing system 18, and, datasecurity 34.

[0014] Many image visualization and processing tasks (such as volumerendering) consist of multiple interactive sub-tasks. For example,visualizing a dataset consists of at least two steps (subtasks): 1)generating a processed image to be displayed; 2) displaying the image.In a client/server-based image processing system, some subtasks areperformed by the client and the other by the server. Using the aboveexample, generating the processed image to be displayed can be performedin entirety on the server, or, partially on the server and partially onthe client. Displaying the processed image is performed on the client.

[0015] Referring now to FIG. 2, a system is shown wherein, ascontemplated in the present invention, several receiving stations 300a-e have access over a network 200 to a transmitting station. Each ofthe receiving stations 300 a-e are structured similarly to the receivingstation 300 shown in FIG. 1. The transmitting station may be consideredthe server and the receiving stations the clients to use theclient/server terminology.

[0016] Referring now to FIG. 3, a flow chart representing stepsperformed according to one preferred embodiment is shown. At step 401,one or more users initiate a session by logging in to the server fromone of the receiving stations 300 a-e. At step 402, one of the logged inusers issues a command to form a conference, and identifies a list ofusers who may participate in the conference. The user who initiates theconference, e.g., the user at receiving station 300 a shown in FIG. 2,may be designated as the conference “driver” by default. The conferencedriver may be, for example, a consulting radiologist. Other designatedconference participants may join the conference. Alternatively, thedriver may review the list of users logged in and select persons toparticipate in the conference. The other participants may be, forexample, 3D technologists, other radiologists, referring physicians, orother healthcare personnel. The driver has the ability to accept orreject a request to join. Alternatively, the driver may designate thatthe conference is “open,” i.e., that other users may join in without anexpress authorization being made by the driver.

[0017] At step 403, the driver initiates a processing command from theclient side. In a preferred operation, the driver, using interface 32,specifies: 1) at least one image data set to be visualized; 2) at leastone data rendering method to be used; 3) the rendering parameters usedby each rendering method, 4) data compression parameters, and 5) thedata transmission parameters for controlling data transmission overnetwork 200. Examples of state parameters are provided in FIG. 4. Inparticular, the driver may, via user interface 32, adjust renderingparameters, e.g., viewpoint, spatial region, and value range of the datato be rendered, and other settings. The techniques for setting andadjusting these parameters include 1) using preset protocols for sometypical settings; 2) inputting a specific setting with a keyboard, amouse and/or other input devices; and/or 3) interactive navigation usinga mouse, a trackball, a joystick, a keyboard and/or other navigatingdevices. This driver may, via user interface 32, edit (includingprocess) patient data, e.g., remove the bone structures, in a mannersimilar to the current volume data rendering/visualization systems. Withthe teleradiology system of the invention, driver can, via userinterface 32, define and adjust data rendering methods and parameters,control what is to be rendered, transmitted and visualized next, andeventually obtain the final rendering result. A central processingsystem 24 on the driver's receiving station receives and validates thedriver's request. The central processing system 24 then issues therequest, which is sent via send request 22 to transmitting station 100through network 200.

[0018] At step 404, the central processing system 18 on the transmittingstation 100 receives the request via receive request 20. Coordinated bycentral processing system 18, volume data rendering generator 12accesses from image data source 10 the image data set which the user hasspecified, and then generates the data rendering result based on thedata rendering method and parameters which the user has specified. Therendering result may be a 2D image, much smaller in size than theoriginal data set.

[0019] At step 405, the data transmitter 16 on transmitting station 100transmits the compressed data to data receiver 26 on receiving stations300 a-e which have sent a request for image data, i.e., on-demand, vianetwork 200 based on data transmission parameters which the user hasspecified. The on-demand feature of the present invention will bedescribe further in connection with FIG. 5. For the teleradiology systemof the invention, the preferred transmission medium (i.e., network 200)may be an intranet, the Internet (including the Internet2) or a directdial-up using a telephone line with a modem. The preferred datatransmission protocol is the standard TCP/IP, although the method may beadapted to accommodate other protocols. Furthermore, for sometransmission media (e.g., the Internet2), user 400 can control certainaspects (e.g., the priority level, the speed) of data transmission byselecting transmission parameters via user interface 32.

[0020] At step 406, the central processing systems 24 of the variousreceiving stations 300 a-e coordinate the client-side processing. Ifneeded, data decompressor 28 decompresses (or restores) the renderingresult. The central processing system 24 may also perform further imageprocessing and operations. The processing is performed in which thefinal image is computed based on the field of view and the imagewindow/level (i.e., brightness/contrast) settings currently prescribedby the conference driver.

[0021] At step 407, the display systems 30 at receiving stations 300 a-edisplay the computed image and other parameters. Via user interface 32,the driver may further modify parameters, including 1) the image dataset to be visualized, 2) the data rendering method to be used, 3) therendering parameters used, and 4) the data transmission parameters used.This process goes on until a satisfactory rendering and visualizationresult is obtained.

[0022] The set of image processing and display parameters, collectivelycalled state parameters, keep track of the effect of image processing,performed either at the server or at a client, and if needed,synchronize the display (viewing) of multiple users. Examples of stateparameters are given in FIG. 4. Each time when a new subtask isperformed, this set of the state parameters is updated at the server.Any further image processing and display task will be performed based onthis set of updated state parameters.

[0023] In one embodiment, the resulting images are “pulled” to theclients from the server. When a client with the driver authorizationprescribes an operation and regardless of whether this operation isperformed on the client, the server, or the both, the state parameterswill be updated on both the server and the driving client to reflect theresultant change due to this operation. Other clients periodicallycompare their local copy of the state parameters with the copy on theserver. If some differences are found that require updating the localdisplay, that client will issue the update request. Again, depending onthe division of subtasks, some requests are fulfilled by the clientonly, while the others require that the server sends updatedimage/information.

[0024] Referring now to FIG. 5, the steps involved in state parameterupdating will be described. State parameter updating is controlled bythe client-side conferencing software running on receiving stations 300a-e. At step 501, a check is made with a system clock, or another timingsource, to determine whether the amount of time that has elapsed sincethe last state parameter update, Δt, is equal to a predetermined timingparameter, P_(t). which determines the frequency with which the stateparameters are updated. If Δt≧P_(t), then step 502 is performed. IfΔt<P_(t), then control returns to the beginning of the routine. Forexample, P_(t) may be 0.25 to 0.5 seconds. At step 502, one of thereceiving station 300 sends a request to the transmitting station forcurrent state parameters associated with the current conferencingsession. At step 503, the receiving station 300 compares the stateparameters which have been stored locally to the state parameters thatare received from the transmitting station after the request made instep 502. If the client state parameters and the server state parametersare equal, then control returns to the beginning of the routine. If thetwo sets of parameters are not equal, this implies that additionalsubtasks have been specified by the conference driver, and the routineproceeds to step 504. At step 504, the client sends a request for newimage data if the parameters that have changed indicate that new imagedata has been generated. On the other hand, if only state parametersrelating to brightness or contrast level, for example, are changed, thenno new image data need be requested, because this change can beprocessed on the data already stored at the client. At step 505, theclient state parameters are set equal to the updated server stateparameters. At step 506, At is set equal to zero.

[0025] What has just been described is an on-demand image transmissionmethod. Unlike the existing conference systems, image transmissionoccurs only when needed, and therefore, the network utilizationefficiency is greatly improved.

[0026] In an alternative embodiment, a “push” implementation isutilized. In the push implementation, state parameters are transmittedto the clients whenever they are changed. Also, new image data istransmitted if, as described above, the change in the state parametersrequired new server-side image processing.

[0027] In another alternative embodiment, all remote conferenceparticipants may have already had the copy of the same data set on eachof their local disk. This may be the case for training or educationalapplications in which a standard set of data is utilized. In this case,no image data transmission is required over the network. Based on thestate parameters maintained on the server, the conferencing softwarerunning on each participant's computer will generate the new image usingthe local copy of the data and local computing resources and willsynchronize the image display. This embodiment is useful when theconference participants only have relatively-narrow bandwidthconnection, such as a phone line, which is adequate to communicate thestate parameters interactively, but not adequate for transmitting bigdata files, such as images, at rate allowing real time interaction.Updated state parameters in this embodiment may be transmitted to theclients either in a push implementation or a pull implementation.

[0028] As part of the preferred embodiment, any participant in aconference may request to become the driver. Upon approval from thecurrent driver, the driver privilege may be switched to the requestingparticipant. The new driver will then have the full control of theimage/data set under study, i.e., the ability to define new stateparameters. The new driver, e.g., a surgeon, may fine tune the 3D modelor other parameters to achieve the best view for his intendedapplication.

[0029] The present invention may also be applied to multi-center trialstudies, when constant communication of comprehensive informationincluding images and data are needed between multiple participants. Oneexample is a Magnetic Resonance Angiography (MRA) multi-center trial. Inan MRA study, a 3D volumetric data set, comprised of a stack of 2Dimages, is acquired. This 3D volumetric data set is processed to extractthe vascular structure, while minimizing the interference of otherunwanted structures. In order to select the highest quality protocolsand design the most effective trial, the participants need to view notonly the acquisition protocol and the original 2D images, but also theprocessed 3D MRA images in detail.

[0030] The existing multi-center trial procedures face severalchallenges. First, in order to reach consensus on trial protocols,principle investigators from participating institutions may need totravel to different locations multiple times, making this process timeconsuming and expensive. Second, the current procedure of siteselection, training, and trial monitoring require frequent travel by thetrial monitors to various participating sites, making this processheavily dependent on the trial monitors' travel schedule andavailability. Third, the current process calls for transferring of allthe patient data/images to a centralized position, demanding significantamount of pre-work to modify studies headers and preserve patientprivacy.

[0031] The present invention provides an optimized method formulti-center trial management using the teleradiology conferencingtechnology. This method is designed to optimize the workflow andmanagement of various tasks, such as protocol selection,training/education, trial monitoring, and data management for expertreading.

[0032] The steps for future multi-center trial management using thepresent invention include:

[0033] 1) Using the teleradiology conferencing techniques describedherein to choose a trial protocol;

[0034] 2) Subsequently using the training embodiment of theteleradiology conferencing techniques described herein to conductinteractive conferences hosted by the sites experienced in the selectedprotocols to provide training/education to other participating sitesusing the mechanism described in the above section;

[0035] 3) Using the teleradiology conferencing techniques describedherein to conduct interactive conferences between the trial monitor andindividual participating sites to review images, in order to assurequality and compliance during trial process;

[0036] 4) Using the teleradiology techniques described in ourapplication, U.S. Ser. No. 09/434,088, and the present invention toallow an expert reader to remotely review, and interactively process ifneeded, 2D/3D image sets store at centralized or disparate locations,without physically transmitting the entire image sets; and

[0037] 5) Reporting Expert reader will report blind read results usingthe integrated reporting tools provided by a system based on the presentinvention.

[0038] While the present invention has been described in its preferredembodiments, it is understood that the words which have been used arewords of description, rather than limitation, and that changes may bemade without departing from the true scope and spirit of the inventionin its broader aspects. Thus, the scope of the present invention isdefined by the claims that follow.

What we claim is:
 1. A system for remote manipulation of image datacomprising: a telecommunications network; an image data storage library;an image processing server coupled to the telecommunications network andfurther coupled to the image data storage library; and a plurality ofreceiving stations coupled to the telecommunications network, each ofthe plurality of receiving stations having a memory for storing localcopies of state parameters, wherein at least one of the receivingstations transmits state parameters through the telecommunicationsnetwork to the image processing server, and wherein the image processingserver receives image data from the image data storage library andprocesses the image data in accordance with the received stateparameters, and wherein the image processing server transmits processedimage data through the telecommunications network to the receivingstation.
 2. The system of claim 1 wherein at least one of the receivingstations transmits a request for processed image data through thetelecommunications network and wherein the image processing servertransmits processed image data to the receiving station through thetelecommunications network in response to the request.
 3. The system ofclaim 1 wherein the image processing server transmits processed imageddata to at least one receiving station upon the completion of theprocessing of image data.
 4. The system of claim 1 wherein a first oneof the plurality of receiving stations includes a user interface meansfor altering the local copy of state parameters and a means fortransmitting the local copy of the state parameters, and wherein thefirst one of the plurality of receiving stations transmits a copy of thelocal copy of the state parameters through the telecommunicationsnetwork to the image processing server.
 5. The system of claim 1 whereina first one of the plurality of receiving stations includes a userinterface means for altering the local copy of state parameters and ameans for transmitting the local copy of the state parameters, andwherein the first one of the plurality of receiving stations transmits acopy of the local copy of the state parameters through thetelecommunications network to the image processing server and wherein atleast one other of the plurality of receiving stations receives thestate parameters through the telecommunications network from the imageprocessing server and stores a local copy in the memory of the at leastone other of the plurality of receiving stations.
 6. The system of claim4 wherein the at least one other receiving station transmits a requestfor processed image data through the telecommunications network andwherein the image processing server transmits processed image data tothe receiving station through the telecommunications network in responseto the request.
 7. The system of claim 4 wherein the image processingserver transmits processed imaged data to the at least one otherreceiving station upon the completion of the processing of image data.8. The system of claim 4 further comprising means for authorizing onlythe first one of the plurality of receiving stations to alter andtransmit the local copy of state parameters, and means for removing theauthorization from the first one of the plurality of receiving stationsand granting the authorization to another one of the plurality ofreceiving stations.
 9. The system of claim 1 further comprising adisplay means coupled to a user input device, the user input deviceincluding a means for manipulating the movement of a cursor displayed onthe display means and a means for causing text and graphics to bedisplayed on the display means, wherein the state parameters includeparameters indicating the location of the cursor on the display meansand the location and content of text and graphics on the display means.10. The system of claim 1 wherein the image data are comprised ofvolumetric data.
 11. A system for remote manipulation of image datacomprising: a telecommunications network; an image data storage library;an image processing server coupled to the telecommunications network andfurther coupled to the image data storage library; and a plurality ofreceiving stations coupled to the telecommunications network; whereinthe image processing server includes a first memory for storing a serverset of state parameters, a server-side machine-readable medium, and aserver-side processor that executes a first program stored in theserver-side machine-readable medium, the first program causing theserver-side processor to perform the steps of: controlling the receptionof an update set of state parameters over the telecommunicationsnetwork; controlling the determination of whether the received updateset of state parameters differs from the server set of state parametersin a manner which requires new processing of the image data; controllingthe processing of image data according to the update set of stateparameters; controlling the transmission the update set of stateparameters from the image processing server to the receiving stations;and controlling the transmission of new image data from the imageprocessing server to the receiving stations if the update set of stateparameters required processing of image data at the image processingserver; and wherein the plurality of receiving stations include a secondmemory for storing a local set of state parameters, a client-sidemachine-readable medium, and a client-side processor that executes asecond program stored in the client-side machine-readable medium, thesecond program causing the client-side processor to perform the stepsof: controlling the transmission of a request for new state parametersto the image processing server through the telecommunications network;controlling the reception of state parameters from the image processingserver over the telecommunications network; controlling thedetermination of whether the received state parameters differ from thelocal set of state parameters and whether the received state parametersrequire non-local processing of image data; and controlling thetransmission of a request for updated image data from the receivingstation to the image processing server if a determination is made in thedetermining step that the received state parameters require non-localprocessing of image data.
 12. The system of claim 11 wherein at leastone of the plurality of receiving stations includes a user interfacemeans for altering a set of state parameters stored in the memory and atransmission means coupled to the telecommunications network fortransmitting the set of state parameters stored in the memory.
 13. Thesystem of claim 11 wherein at least one of the receiving stationstransmits a request for processed image data through thetelecommunications network and wherein the image processing servertransmits processed image data to the receiving station through thetelecommunications network in response to the request.
 14. The system ofclaim 11 wherein the image processing server transmits processed imageddata to at least one receiving station upon the completion of theprocessing of image data.
 15. A system for remote manipulation of imagedata comprising: a telecommunications network; a communications servercoupled to the telecommunications network; a plurality of receivingstations, the receiving stations including a first memory for storing alocal set of state parameters, a first client-side machine-readablemedium containing a pre-stored set of image data, a second client-sidemachine-readable medium, and a client-side processor that executes aprogram stored in the second machine-readable medium, the programcausing the processor to perform the steps of: controlling thetransmission of a request for new state parameters to the communicationserver through the telecommunications network; controlling the receptionof state parameters from the communications server over thetelecommunications network; controlling the determination of whether thereceived state parameters differ from the local set of state parameters;and controlling the processing of the pre-stored image data based on thereceived state parameters; wherein the communications server includes asecond memory for storing a server set of state parameters, aserver-side machine-readable medium, and a server-side processor thatexecutes a second program stored in the third machine-readable medium,the second program causing the second processor to perform the steps of:controlling the reception by the communications server of an update setof state parameters over the telecommunications network; and controllingthe transmission the update set of state parameters from thecommunications server to the receiving stations.
 16. The system of claim15 wherein at least one of the plurality of receiving stations includesa user interface means for altering a set of state parameters stored inthe memory and a transmission means coupled to the telecommunicationsnetwork for transmitting the set of state parameters stored in thememory.
 17. The system of claim 15 wherein at least one of the receivingstations processes image data stored in the local memory according tothe received set of state parameters.